feat: add Inspect AI integration package (learning-commons-inspect-scorers) - #102
Draft
adnanrhussain wants to merge 8 commits into
Draft
feat: add Inspect AI integration package (learning-commons-inspect-scorers)#102adnanrhussain wants to merge 8 commits into
adnanrhussain wants to merge 8 commits into
Conversation
…el injection Introduces three types to sdks/python: - LLMGeneratorProtocol (typing.Protocol) — structural interface for injecting any LLM backend into evaluators without a hard framework dependency - LLMResponse (NamedTuple) — structured response aligned with OTel GenAI semconv (content, model, input_tokens, output_tokens) - GenerateConfig (dataclass) — temperature and max_tokens passthrough Refactors BaseEvaluator.execute_prompt_chain_step to accept an optional llm_provider: LLMGeneratorProtocol. When set, the protocol path formats the LangChain template to extract system/human strings, delegates the LLM call to the injected provider, and parses the JSON response via Pydantic directly. The existing LangChain path is unchanged and remains the default. Also improves _strip_json_fences to use JSONDecoder.raw_decode, correctly handling trailing prose and multiple JSON objects in LLM responses. Also adds *.egg-info/, dist/, build/, logs/ to root .gitignore.
…_step Adds TestExecutePromptChainStepProtocolPath (13 tests) covering the llm_provider injection path in BaseEvaluator.execute_prompt_chain_step: - Raw string return when parser_output_type=None - Clean JSON parse - Markdown fence stripping - Trailing prose stripping (JSON followed by explanation text) - Leading prose stripping (prose before JSON) - json_dict_normalizer path - Non-dict JSON raises OutputValidationError on normalizer path - Malformed JSON raises OutputValidationError - Schema mismatch raises OutputValidationError - Token usage recorded in step extras and total_token_usage - Token usage absent when LLMResponse has None tokens - Provider RuntimeError wrapped as APIError - EvaluatorError from provider re-raised unchanged - KeyboardInterrupt from provider propagated
- Revert version bump to 0.2.0 (release-please handles this on merge) - Add model field to GenerateConfig so adapters can see which model the evaluator expects without reaching into prompt_settings - Move template.aformat_messages() inside try block so missing template variables raise EvaluatorError rather than bare KeyError - Add ValueError when human_str is empty; DEBUG log when system_str is empty - Extract _parse_json_output() helper to deduplicate JSON parsing and error wrapping between the protocol and LangChain paths - Add tests: assert adapter.generate() called with correct system/human, human-only template passes empty system string, missing variable raises EvaluatorError
adnanrhussain
force-pushed
the
ahussain/inspect-integration
branch
from
June 23, 2026 04:04
1fcc67c to
6bc1c96
Compare
adnanrhussain
force-pushed
the
ahussain/sdk-llm-protocol
branch
from
July 28, 2026 22:34
7d75e0d to
491f581
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
integrations/inspect-python(learning-commons-inspect-scorers) — the Inspect AI integration for the LC evaluators SDK.This is split out from the combined integrations PR (#101) so the Inspect path — the only one with a real consumer (
edu-panda-skill-harness) — can be validated and merged independently. The speculative observability integrations (Arize, Langfuse, Braintrust) remain on #101 to be revisited per-vendor once each is validated against a real account.What's here
InspectModelAdapter— wraps Inspect'sget_model()to satisfyLLMGeneratorProtocol, so the GLA evaluator runs through Inspect's own model system (no separate API keys).gla_scorer()— an Inspect@scorerfor grade-level appropriateness. Readstarget_gradefrom sample metadata, scorescompletionorartifacts, returnsCORRECT/INCORRECTwithScore.unscored()for skip/error paths.inspect score --scorer learning_commons_inspect_scorers/gla_scorer.Review fixes already applied
inspect-ai>=0.3.214—Score.unscored()(used in every skip/error path) was added in 0.3.214; the previous>=0.3.2bound wouldAttributeErrorat runtime on older installs.Test plan
eval()withmockllm/model)integrations/inspect-python/**changes.🤖 Generated with Claude Code